Storage Management for High Energy Physics Applications

In many scientific domains large volumes of data are often generated by experimental devices or simulation programs. Examples are atmospheric data transmitted by satellites, climate modeling simulations, and high energy physics experiments. The volumes of data may reach hundreds of terabytes and therefore it is impractical to store them on disk systems. Rather they are stored on robotic tape systems that are managed by some mass storage system (MSS). A major bottleneck in analyzing the simulated/collected data is the retrieval of subsets from the tertiary storage system. This bottleneck results from the fact that the requested subsets are spread over many tape volumes, because the data are stored as files on tapes according to a predetermined order, usually according to the order they are generated. In this paper we describe the architecture and implementation of a Storage Manager designed to support the ordering of the data on tapes to optimize access patterns to the data. We also describe additional optimization opportunities to improve access time. The system is being built for a High Energy Physics experiment scheduled to go on-line in a year.